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Abstract: Little information exists regarding mental health 
and special needs related to American Indian and Alaska 
Native (AI/AN) families. In this article we emphasize the use 
of oral tradition during the Circles of Care initiative, which 
M/as essential in understanding cultural history and historical 
trauma of AI/ANs while giving a greater understanding of an 
AI/AN-hased definition of serious emotional disturbance. 

The success of these methods serves as a template for 
improving systems of care and may be useful in evaluation 
among a wide range of ethnic communities. 


You have noticed that everything an I ndian does is in a circle, 
and that is because the Power of the World always works in 
circles, and everything tries to be round.... The Sky is round, 
and I have heard that the earth is round like a ball, and so 
are all the stars. The wind, in its greatest power, whirls. 
Birds make their nest in circles, for theirs is the same religion 
as ours... Even the seasons form a great circle in their 
changing, and always come back again to where they were. 
The life of a man is a circle from childhood to childhood, 
and so it is in everything where power moves. - Black Elk, 
Oglala Sioux Holy Man 


Clearly there is great diversity among American I ndian and Alaska 
Native (AI/AN) communities in both pre-colonial and postcolonial contexts, 
and this diversity was evident in the broad range of cultural, organizational, 
and geographic settings of the Circles of Care (CoC) grantee communities. 
How mental health and other special needs are understood and have impacted 
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AI/AN families is an area of limited literature. Historically, diagnosis and 
treatment of Al /AN people has been influenced by the interpretations of non- 
AI/AN peoples. The CoC initiative represented one of the first opportunities 
for AI/AN families and their communities to have substantial input into the 
design of services to address the needs of their children, from their own 
understandings of these needs. This fact alone makes the contributions of 
this special issue invaluable. 

In addition to presenting an overview of the evaluation research 
findings, this special issue describes the CoC evaluation process, and presents 
practical planning ideas for groups considering evaluation of similar projects 
that are culturally based and culturally driven. The diversity of tribes and the 
community milieu that AI/AN families live in today must be viewed within 
each unique and local tribal community context. Often, there is the tendency 
to look for and to pose solutions that apply across tribes and regions. I n the 
CoC experience, understanding and working within the community context, 
it became apparent through both the evaluation findings and the diversity of 
planning effort solutions (documented in the monograph produced by the 
National Indian Child Welfare Association which was still in preparation at 
the time this paper was written) that solutions to challenges in AI/AN 
communities must be specific to local needs and service system ecologies 
and be culturally relevant. 

I n the understanding and search for community-based solutions to 
any issue in AI/AN communities, a discussion of history and trauma is 
important. Historically, one very important factor that must be considered is 
the legacy of mistrust that AI/AN people have for government, public health 
officials, and the intentions of Western health and mental health providers. 
Sadly, this mistrust is well founded and examples can be found in each of the 
tribal histories, from the "gifts" of blankets infected with smallpox to the 
sterilization abuses of the seventies (J ennings & Asetoyer, 1996; Taivi, 1997). 
This mistrust extends to health and mental health services, where issues of 
confidentiality have been questioned, quality of care has been debated, 
discrimination found, and depersonalization issues prominent. These are 
particularly important factors to consider for providers in tight-knit 
communities, where rumors and innuendo are common. Clearly, this history 
of distrust prevents many AI/AN families from seeking diagnosis, assistance, 
and mental health services. 

Certainly, AI/AN communities have experienced a wide variety of 
community problems and challenges, including the impact of intergenerational 
trauma. Although the construct of intergenerational trauma has long been 
known to healers and Elders in AI/AN communities, and has been reported 
in clinical studies of j ewish Holocaust survivors (Yehuda, Schmeidler, Giller, 
Siever, & Binder-Brynes, 1998; Yehuda, Schmeidler, Wainberg, Binder-Brynes, 
& Duvdevani, 1998), it is new to many disciplines and should therefore be 
discussed briefly. 
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Historical trauma is recognized in AI/AN circles as a result of 
colonization that brought oppressive federal policy, forced change, and 
disparate treatment of AI/ANs. Historical trauma or intergenerational trauma, 
then, is offered as a paradigm to explain, in part, many of the contemporary 
social issues that have plagued AI/AN communities (Brave Heart & DeBruyn, 
1998; Weaver & Brave Heart, 1999). These historical factors must be 
considered when planning services for AI/AN communities. I ndeed, these 
factors were a major consideration throughout the CoC initiative. 

Similarly, these factors were incorporated into the content of this 
Special Issue, which has sought to provide an in-depth description of the 
program evaluation component of the CoC initiative. For example, the Special 
I ssue's second paper introduces the concept of the life cycle of the evaluation 
process (Bess & Allen, 2004). Certainly evaluation is not new to AI/AN 
people. Rather, evaluation is simply the accurate telling of a very important 
story. AI/AN groups have relied on the oral tradition of story telling for 
centuries. The stories were used to teach, to provide guidance and direction 
to tribal members, to maintain and perpetuate tribal stories, legends, customs 
and values, to keep an accurate accounting of the tribe's life cycle and history, 
and to shape the future of the tribe. I n 1964, Helen Cordero of the Cochiti 
Pueblo created the first contemporary storyteller clay figure in honor of her 
grandfather, Santiago Quintana, a storyteller and one of the first people 
from the Pueblo to work extensively with both researchers and ethnographers. 
Thus evaluation, like storytelling, is used to teach, to provide guidance and 
direction to key stakeholders, to maintain and perpetuate program stories, 
customs and values, and to keep an accurate accounting of the program's 
life cycle, history, and to shape its future. 

It was the hope of the numerous authors of this Special Issue to 
accurately tell the story of the life cycle of the CoC evaluation, beginning with 
the recognition that planning was necessary, through the process of the 
needs assessment, service system description, outcome assessment, and 
feasibility assessment process. The outcomes realized through this process 
were very rich and led some tribal groups to pursue the SAMHSA 
Comprehensive Community Mental Health Services for Children and their 
Families grant, as well as other grant mechanisms. This funding enabled 
CoC grantees to implement a system of care that came out of their carefully 
developed planning process, or to pursue the new beginnings of separate 
service components. 

It was also important, and specific to this initiative, to recognize 
tribal needs for more culturally grounded definitions for the label of serious 
emotional disturbance (SED). While both federal and state governments 
have specific defining terminology to describe SED, there was some difficulty 
in accepting the terminology on the part of AI/AN communities in the CoC 
initiative. Out of this discomfort, some grantee communities created their 
own definitions that reflected higher levels of cultural congruence. 
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Vignette: An emotional disturbance is a temporary 
disharmony, often involving the family, school, and 
community, which may affect the mental, physical, spiritual, 
and/or emotional well-being of its members. Healing the 
disharmony returns us to the basic belief that... "VILA 
VLTHLEHA VT HOLITOPA" Children are Precious. - Choctaw 
Nation of Oklahoma's definition for children with serious 
emotional disturbances. 

Throughout this paper as well as this Special I ssue, we have focused 
on the common experiences reflected in all of the grantee sites, rather than 
individual grantee stories. I n keeping with this spirit, we offer conclusions 
and recommendations that arise from the collective CoC experience in the 
hope that telling this story will be useful in the evaluation of similar projects 
in AI/AN communities. We believe that the participatory approach chosen by 
those who administered the CoC initiative allowed grantees to learn from 
their communities of concern, from one another, and to gain support both 
from their communities and their shared experiences. Recommendations 
will be offered that we hope will be practical to use in evaluation work with 
other ethnic minority communities. I n order to provide a clear context for 
understanding the recommendations offered, it would be helpful to first review 
the original goals of the CoC initiative. 

The Goals of the Circles of Care I nitiative 

The CoC initiative offered funding to plan, design, and assess the 
feasibility of implementing a culturally appropriate mental health service model 
for AI/AN children with SED and their families. The four goals of the CoC 
Guidance for Applicants (GFA) were to: 

1. Support the development of mental health service delivery models that 
are designed by AI/AN communities to achieve outcomes for their children 
that they choose for themselves; 

2. Position tribes, tribal groups, or villages advantageously for future service 
system implementation and development; 

3. Strengthen tribes', tribal groups, or villages capacity to evaluate their 
own service system's effectiveness, and 

4. Develop a body of knowledge to assist tribal, tribal group, village, and 
other policy-makers and program planners for all child-serving systems in 
improving systems of care for the AI/AN population overall (CoC GFA, p.9). 

By way of summary, we discuss below how the evaluation effort 
contributed to each of these goals. 



THE CIRCLES OF CARE EVALUATION 


143 


Support the Development of Mental Health Service Delivery 

This concept, which provided for the creation of this initiative, was 
really quite innovative in its approach. Generally, GFAs are by nature quite 
specific in structuring what is appropriate for funding and exactly what types 
of activities are allowable. However, this GFA allowed tribes to actively identify 
a definition for SED, to develop the strategic planning approach as locally 
appropriate in the development of a practical system of care model and to 
examine the feasibility of this model; all conducted within the context of the 
culture of the participating tribes. 1 1 allowed latitude in selecting the types of 
outcome that AI/AN families and providers wanted for their children. It 
called for community involvement and included families and important AI/AN 
community and tribal leaders in the planning, as well as in grantee workshops 
and conferences. Although the GFA provided a specific structure for applicants, 
many of the decisions for the content and context of a service model were in 
the hands of each tribal grantee. 


Position Grantees Advantageousiy for Future Service System 
I mpiementation 

In implementing CoC, the Substance Abuse and Mental Health 
Services Administration and its partner agencies (the Indian Health Service, 
the Department of J ustice, and the National I nstitute of Mental Health) were 
invested in the success of both the initiative and in the tribal grantees. Thus, 
a key factor in the success of CoC was the administration of the overall 
initiative. The partner agencies experienced challenges in pulling such diverse 
tribal groups together, and yet the leadership was strong and the direction 
clear. For example, the National contractual evaluation requirements were 
substantive and difficult for tribes to implement initially. Understanding on 
the part of the contractor was helpful in overcoming this situation. Many 
tribes were not aware of the types of technical assistance that were available 
by the technicial assistance contractor and it was the second or third year 
before this information was utilized by many tribes. However, the one on 
one assignment of a technical assistance specialist for each tribe met this 
challenge very well. Both contractors worked diligently to make this 
partnership a success. These funding agencies, their representatives, and 
the leadership of the initiative demonstrated a great deal of support and 
involvement throughout the life of the initiative. 

Grantee meetings were held two to four times a year. These periodic 
meetings provided a strong networking environment wherein grantees could 
share experiences and ideas, learn from one another, and gather strength 
not only from the leadership of the initiative, but from other grantees as 
well. This communication and support generated like behavior for grantees 
and their individual sties. Results of the planning process included 
requirements for community involvement, evaluation methods for gathering 
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data (including the use of focus groups, surveys and interviews), community 
dissemination of data, and often, community and consumer involvement in 
the interpretation and reporting of the evaluation data. Tribes were then 
allowed to place these results into more culturally relevant contexts. For 
example, some tribes gathered information during GONA's (Gathering of Native 
Americans), allowing them to develop strategies consistent with each tribal 
culture. In addition. Healing Circles, rather than focus groups, allowed 
community members a safe environment for discussion of frustrations with 
the current system and needs for a future system. Finally, dissemination of 
evaluation data often occurred through cultural venues, such as tribal 
newspapers, tribal radio stations, tribal councils, and community-wide 
meetings. 

These types of culturally-based activities fostered high levels of 
communication and information exchange within many of the grantee sites, 
resulting in strong community investment in the planning process and the 
eventual outcome of services to families. Moreover, the evaluation gathered 
significant amounts of information on need, service disparity, feasibility of 
their strategic plan, and locally defined outcomes. This information gave 
grantees many important tools, for use in developing competitive grant 
proposals for service system implementation. 1 1 also ensured that strategies 
were based on local needs, and community involvement was a key factor in 
development and ownership of the strategies. The combination of data, 
community involvement, and local investment in the strategic plan positioned 
many of the CoC grantees quite advantageously for the development of 
competitive grant proposals for future service system implementation and 
development. Even more importantly, these factors contributed to the 
development of strategic plans that actually fit the needs of the AI/AN 
communities, and were acceptable because they were grounded in the local 
contexts and culture of the participating tribes and urban communities. 


Strengthen Tribal Capacity for Evaluation of Service Systems 

A decision was made prior to release of the GFA to provide funding 
for two technical assistance teams to be utilized by the CoC grantees. This 
proved to be a wise decision because these technical assistance teams 
provided an informed and valuable body of information to the grantees. 
Technical assistance for the evaluation process was provided by the CoC 
Evaluation Technical Assistance Center (CoCETAC), National Center for 
American Indian and Alaska Native Mental Health Research, University of 
Colorado Health Sciences Center. Although at times this was a challenging 
process, grantee groups were provided training opportunities at each 
conference and in addition, met with experienced consultants who advised 
and taught methods of evaluation. Most importantly, these consultants, who 
truly served as partners in the evaluation process, provided grantees with a 
model for evaluation that was practical, useful, and culturally appropriate. 
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The technical assistance partners emphasized frequently that they were 
learning from the grantees, just as the grantees were learning from them. 
Their peer review of efforts and products set standards for quality control 
that greatly benefited the grantees. While reviewing work, CoCETAC staff 
were timely in offering support, feedback, and vital information to grantees. 

I mportantly, they listened thoughtfully to grantee comments and concerns 
about evaluation. This reciprocity resulted in a rich transfer of contextually 
relevant skills that strengthened tribal capacity to evaluate their own service 
systems in a culturally congruent manner. 


Development of Knowledge to I mprove Tribal Systems of Care 

The sustained efforts of the evaluation process developed extensive 
data sets, and subsequently a significant body of knowledge within each 
grantee community on children's services. Overviews of these resulting 
data have appeared in this Special Issue. Perhaps unique to CoC was the 
extent of the sharing of knowledge between multiple grantee sites, 
administrative leadership and grantees, and technical assistance teams and 
grantees. This information sharing provided an even broader and stronger 
knowledge base from which to build systems of care within the context of 
each tribal community. The emerging knowledge base that resulted from 
the evaluation work was crucial to the planning process and provided a 
model of supportive advocacy for future efforts. 

The outcomes from these four goals culminated in the overall goal 
of CoC: to plan, design, and assess the feasibility of implementing a culturally 
appropriate mental health service model for AI/AN children with SED and 
their families. The planning effort, informed by the evaluation process, 
resulted in the development of a very unique, culturally grounded, and 
extensive body of knowledge within each community and across tribal entities. 
This body of knowledge will continue to assist tribes, villages, and other 
policy-makers and program planners of child-serving systems in improving 
systems of care for AI/AN populations. Moreover, these tribal communities 
now possess an evaluative product and accompanying knowledge base from 
which they can draw in the exercise of self-determination. Tribal communities 
who participated in CoC can respond to new competitive initiatives with 
comprehensive data sets to support their vision, and conversely, can respond 
to inappropriately posed solutions, models of services, and outcomes with 
quite sophisticated data to the contrary. 

Rural/ Urban Differences 

It seems imperative to make certain that the recommendations 
generated differentiate between rural/ reservation, village, and urban tribal 
communities. Strategies, methodologies and levels of resources varied 
considerably within these groups. Three and possibly four separate systems 
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exist within tribal systems of care, based upon geography, population 
dispersion, and political organization (urban, village, rural and reservation). 
Though they shared a similar goal within CoC for using evaluation data as 
part of the planning process, the evaluation and planning processes functioned 
quite differently across these different types of settings. 

As described in Allen and LeMaster (2004) of this special issue, 
urban and rural settings often had different levels of professional and 
traditional resources. We wish to emphasize here that we are not implying 
that one is better than another, rather that there are important differences in 
the nature and access to resources between settings. 

Finally, rural and urban planning and evaluation also organized in 
relation to government entities, as outlined below: 

1. Urban programs had to recognize spheres of influence of various state, 
county, city, and nonprofit programs that served AI/AN children and families 
both the planning process and the evaluation process. 

2. Rural programs had fewer programmatic resources; however rural 
government is smaller and closer to home, and therefore potentially more 
responsive to local community planning and evaluation processes. 

3. Though reservations are generally sovereign, they are compelled to 
collaborate with the state and county governments; however, federally 
recognized tribes seem to have more control over resources. 

4. The element of self-determination reverberates throughout both rural 
and urban settings. 


Rural and urban differences also directly impacted the process of 
evaluation. Almost all of the urban, rural, reservation, and village areas 
utilized evaluation teams generally associated with university systems, 
increasing the number and type of resources available to them. However, 
the planning process for collection of data varied immensely. Travel between 
villages in roadless rural Alaska is costly and time consuming, and thus, 
attending a tribal or village community meeting was typically a two- or three- 
day process. Under these circumstances, conference calls, electronic mail, 
and planning for visits became a key factor in the planning process. Travel in 
rural and reservation areas was a similar challenge for many of the grantees. 
For example, it sometimes took an entire day to travel to a focus group or 
healing circle, conduct the circle and return to home base. Urban areas may 
not have faced the same challenges, but mobility of families and transitional 
relationships increased the challenge of planning for consistent services to 
families in different ways across urban settings. 
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I mpoitance of the Three- Year Grant Process 

There have been past and current attempts by the Substance Abuse 
and Mental Health Services Administration and the I ndian Health Service to 
conduct this type of planning process within two years or even one, yet 
considering all of the elements, politics, and collaboration that must occur, a 
three-year time frame appears more appropriate. Looking back, a full three 
years was required for the planning and evaluation process to occur and in 
some cases agencies requested a fourth year through no-cost extensions. 
The time allotted in this GFA allowed for building of trust and collaboration, 
as well as for consideration of the complexity of the relationships between 
agencies and governments in the system of care within AI/AN communities. 
A three-year time frame also allowed for sharing data, community 
involvement, and the extensive data collection required by the evaluation. 
These were all necessary elements for program success. As noted in the 
previous chapter this special issue on outcomes, there was a rippling effect 
found in the overall outcome review (i.e., one outcome leading to another), 
and the element of time was essential to this process. When one looks back 
over the accomplishments of this initiative (see Table 2, p. 124 in Duclos, 
Phillips, & LeMaster, 2004), it becomes clearer why three full years were 
needed to achieve this degree of success. The primary goal of the CoC 
planning process was to develop or change a system of care to make it more 
culturally competent and culturally responsive, and to build enduring 
collaborations within the service system. To accomplish this feat, even within 
a three- year time frame, was a formidable task. 

Participatory Process and Group Cohesion 

A key factor contributing to the success of this initiative was its 
participatory process. Evaluators, program directors, and community 
stakeholders typically found themselves working together as a close-knit 
team. Much of the planning process was integrated with evaluation, to the 
extent that often one could not really tell which issues were specific to planning 
and which were specific to evaluation. This integration of tasks contributed 
to an open style of communication that in turn enhanced the collaboration of 
evaluation and program staff, consumers, families, and communities in the 
design of the evaluation, data collection activities, and interpretation of 
evaluation results. I n many of the settings, there existed an environment of 
sharing and listening; the willingness to listen and hear was evident, even 
though at times the stories may have been heartbreaking or even invoked 
anger. Often asking the questions was a difficult process because the answers 
were hard to hear. They outlined our failures as a system and were presented 
with a great deal of emotion, since they involved the families’ frustration 
with the system and its failure to provide access to needed services. 



148 


VOLUME 11, NUMBER 2 


The level of communication and sharing that permeated the CoC 
initiative, from beginning to end, resulted in a group cohesion rarely seen in 
other initiatives: from the funders and administrators to the key stakeholders 
and tribal members in the communities. Participants in many of the sites 
found themselves bound by a single goal - to improve services and ultimately 
the welfare of AI/AN children and families. Evaluators were most successful 
when they were not only open and willing to listen, but more importantly, 
were able to engage communities of concern as co-investigators and allow 
the community planning process to lead in the interpretation and understanding 
of the information collected throughout the initiative. 

Conclusions and Recommendations 

A set of basic strategic planning beliefs and recommendations evolved 
naturally over the course of the CoC initiative. These were not explicitly 
stated to the grantees, but over the course of the evaluation technical 
assistance workshops and site visits, the following strategic planning/ 
evaluation principles became evident: ^ 

1. Effective evaluation requires the direct participation of key stakeholders. 

As part of the Government Performance and Results Act of 1993, 
federal agencies are required to consult with stakeholders during strategic 
plan development. I n the context of governmental agencies, this requirement 
might be thought of as a control mechanism to assure that governmental 
agencies are responsive to public interest, rather than to self-interest (Aimee, 
2001) . For systems of care planning, the value of stakeholder consultation is 
even more prominent, not as a control mechanism, but as a means of 
integrating one of the basic principals of systems of care— that community 
input is crucial to effective service— into the strategic planning process. 
Strategic planning for the CoC project was designed to reflect the voices of 
youth. Elders, and families as the driving force. 

2. In effective evaluation, the unique characteristics and needs of different 
tribal entities are recognized, including recognizing tensions between 
traditions, culture, and rural-urban-reservation factors. 

In light of self-determination, stereotypes about AI/ANs, and the 
exclusion of the unique viewpoints and different tribal cultures in written 
historic accounts, AI/AN communities are particularly oriented toward 
understanding that a one-size-fits-all approach to strategic planning or to 
evaluation is inappropriate. The recognition of the tension between the 
general and the specific was best exemplified by the sometimes painful process 
of determining cross-site evaluation domains in CoC that honored the 
commonalities of the grantees, while acknowledging the uniqueness of 
individual tribes. This important consideration will surface in the evaluation 
of any large project across multiple tribal settings. 
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3. Effective evaluation contributions to strategic planning encompass an 
outcome-oriented approach, with outcomes determined by culturally 
appropriate and participatory action research methods. 

Outcomes accountability in mental health strategic planning is not a 
new concept, having been required by PL 94-63 for Community Mental Health 
Centers in 1975. Yet the shift from funding based upon intrinsic good of the 
services toward a results-based accountability has been slow in becoming a 
reality (Hernandez & Hodges, 1998). Within AI/AN communities, top-down 
outcomes, particularly when they originate outside of tribal settings, may be 
thought of as a source of tension, perhaps due to the history of the misuse of 
AI/AN communities for research and the lack of recognition of self- 
determination policies. Yet, over the three years of the CoC process, the 
value of participatory development of outcomes and community-based 
outcomes emerged as a cornerstone of the planning process. This process 
showed how locally developed outcomes could allow for the inclusion of 
culturally appropriate strategies and assist in the development of culturally 
congruent models of care. 

4. Strategic planning processes need to be documented through process 
evaluation. 

Descriptive methodology that fostered the telling of stories of the process 
of community involvement, the empowerment of parents, and the struggles 
to involve key stakeholders emerged as an important effort in the task of 
producing the model, and as a valuable effort in its own right. 

5. Successful evaluation requires the development of evaluator credibility. 

Credibility is earned. It requires the collaborative efforts of staff 
and evaluators to be consistent in their behavior, respectful of communities, 
and to follow through on tasks in predictable ways overtime. Predictability 
assured others of the trustworthiness of the staff and evaluators, and by 
association, the trustworthiness of the project. Credibility was also earned 
by allowing others to create expectations of evaluators that evaluators would 
then seek to fulfill. 

6. Effective evaluation of strategic planning includes a central belief in change 
and change processes. 

Although no change theory or logic model was imposed upon the 
planning process, grantees were encouraged to plan from the standpoint of 
understanding change processes. Community Readiness Assessment 
(Edwards, j umper-Thurman, Plested, Getting, & Swanson, 2000) was one 
model that was a successful tool in several CoC programs. Community 
Readiness and methods employed by other grantees provided important 
baseline information and measures of change. Equally important, these 
processes fostered community involvement, provided valuable qualitative 
information for development of the model, and facilitated the development 
of a structure for intervention and development of action plans. At a primary 
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level, four understandings of change were implicit in the CoC evaluation 
process: 

(a) Change is possible. 

(b) It is important to identify the potential endpoints of the change 
process. 

(c) There are multiple paths leading to each identified endpoint. 

(d) Different programs and agencies move at various paces along these 
paths, and this is a part of the natural process of change. 

1. Effective evaluation in AI/AN communities recognizes that cultural 
competence is not only a desired product of the planning effort, but is also a 
crucial element of the evaluation process itself 

Cultural competence as a product has long been expected, but the 
understanding and implementation of cultural competence in the evaluation 
process was not only complex but also crucial to successful works with Al/ 
AN communities. Working within the culture of the community, respecting 
the community's readiness for change, promoting tribal self-determination 
in evaluation practice, and enacting the formal and informal protocols in 
relationships with groups, families and community members exemplified 
culturally competent evaluation practices. 

8. In effective strategic planning processes, evaluation and project 
development must integrate, though each has a distinct set of responsibilities. 

The symbiotic relationship between evaluation and planning that occurred 
to great benefit within CoC was facilitated, in part, when evaluators became 
not only observers, but also active participants in the process. This required 
flexibility among evaluators, close attention to evaluation ethics, and careful 
and ongoing examination and self-scrutiny regarding roles within the project. 
Evaluation assisted the planning process to the extent that it included and 
valued multiple perspectives through participatory methodologies and a 
commitment to valuing multiple perspectives. Evaluation was successful in 
CoC to the extent that it provided the information the project staff needed to 
plan effectively; and this was facilitated by the extent to which it trained key 
stakeholders in the evaluation process and involved them as co- investigators 
in the design, execution, and data interpretation of the evaluation 

The planning effort informed evaluation by establishing relationships 
with key stakeholders who collaborated in the evaluation process and provided 
information on culturally appropriate methods of approaching evaluation within 
the community context. At its best, evaluation in turn challenged the planning 
process to create a broader, more comprehensive vision of their system of 
care. The planning effort then rendered that vision practical. I n a symbiotic 
way, vision stretched practice, even as practice rooted vision. 

The evaluation model suggested by the integration of project and 
evaluation, and community as co- researcher is a participatory action research 
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approach (Reason & Bradbury, 2001), wherein the evaluator is not simply in 
the auditor or observational role. Particularly for AI/AN communities, the 
potential tensions between the evaluation and project staff are part of a 
mosaic of top-down relationships with government projects and a history of 
negativity associated with agents representing governmental organizations 
such as the Bureau of I ndian Affairs. Referring to systems of care evaluators, 
Hernandez and Hodges ( 1998) cite several pitfalls of the traditional separation 
of evaluation and project development, including adversarial relationships, 
long-terms outcomes with little connection to actual practice, lack of 
communication between the project staff and the evaluator, detachment in 
measurement stemming from a position of scientific objectivity, and a lack of 
useful feedback from evaluation processes. Due to the careful design of 
CoC, these pitfalls were less problematic across the three years of this 
initiative. 

9. In the reporting of evaluation results, consensus or recognition of 
differences may be vaiued more than compromise in many AI/AN 
communities. 

Given the historical issues related to tribes, it is important to note 
that in the context of strategic planning with AI/AN communities, artificially 
homogenizing opposing viewpoints of key stakeholders may not be culturally 
appropriate, and could in fact be viewed as an oppression strategy, given 
that consensus or recognition of differences is often valued more than 
compromise. This is an important overarching recommendation for both 
evaluators and planners to bear in mind in their work. 

10. Evaiuation occurs within a wider socio-poiiticai context in AI/AN 
communities. 

Evaluators will be most successful to the degree that their work 
fosters, rather than hinders tribal self-determination and sovereignty. 
Evaluators need to recognize they work within the context of a history of 
research that has not served AI/AN communities well; hence, credibility must 
be earned. 

11. The process of engagement for the evaiuation is defined by a common 
set of AI/AN values that include the establishment of trust, fiexibiiity, reciprocity, 
suspending judgment, inciusion, and bravery. 

The process of engagement for evaluation first required the 
establishment of trusting relationships between evaluators and participants, 
and could not successfully proceed until this occurred. The process required 
flexibility on the part of evaluators in their approach, methods, and ability to 
adapt to changing circumstances through things learned, including approaches 
that did not work. Reciprocity or the giving back to the community was an 
important requirement for evaluators. They could not be seen as taking 
information from the community without giving back to it. Successful evaluation 
also required suspending judgment and inclusion, which required the 
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acceptance of multiple perspectives. Finally, the process of engagement in 
AI/AN communities included bravery. Bravery surfaced in a willingness and 
trust on the part of consumers to tell their story at no small risk or pain to 
themselves, a willingness of evaluators to do things in a different way and to 
acknowledge publicly their limitations, as well as a willingness on the part of 
policy makers to take on an enormous risk for thinking outside the box, in 
hopes it would better serve AI/AN children and families. 

12. As in other aspects of AI/AN life, spirituai vaiues become an important 
part of the evaiuation process and how the evaiuation is conducted. 

The spirituality of AI/AN people surfaced in specific ways in the 
evaluation process, through the commitment to a superordinate goal. Most 
concretely, this meant that people involved could have lower levels of 
disagreement on specific issues, yet they remained focused and committed 
to the larger task embodied in CoC. Meetings and gatherings began and 
ended with prayer, singing with the drum, circle dancing and asking the 
Creator for guidance. Finally, the presence of family members at the grantee 
meetings, talking about the pain in their hearts, brought spirituality to the 
forefront as well. 


Conclusion 

To its credit, in keeping with the self-determination philosophy of 
CoC, the initiative did not impose a rigid national evaluation plan. Flolden, 
Friedman, and Santiago (2001) note that when a national evaluation plan 
exists, a degree of tension and resistance are natural by-products of its top- 
down nature. Flowever, in the absence of a national evaluation or an imposed 
specific strategic planning model, the CoC initiative at times experienced a 
natural void and a sense of ambiguity. This led to some frustration on the 
part of grantees, yet this is to be expected when self-determination is 
implemented. This void was eventually filled by nine unique and varied local 
solutions to the evaluation and the strategic planning process, with the 
additional benefit of increased community and family involvement, and 
enhanced cultural competence in the approaches. 

The CoC initiative resulted in nine community-specific systems of 
care models for AI/AN children, as evidence of the initiative's strong support 
of grantees in their development of culturally driven service delivery systems. 
Several of the CoC grantees went on to successfully apply for implementation 
of their strategic plans through various grant mechanisms, including the 
SAMFISA Comprehensive Community Mental Flealth Services for Children 
and their Families initiative that were previously out of reach of the planning 
infrastructures of their communities. 

The process of these major accomplishments positioned each tribe 
or urban agency advantageously for future implementation and development 
through other funding sources as well. Each grantee's existing service system 
reported being strengthened by the planning process. 1 1 is our hope that the 
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CoC evaluation process and resulting products described in this publication 
can now be used to assist other tribes and urban AI/AN organizations in 
improving their own systems of care and further, that its methods and 
approaches may be of use in evaluation among other ethnic minority 
communities. 

I n our every deliberation we must consider the impact of 
our decision on the next seven generations - Great Law of 
the Six Nations I roquois Confederacy. 

Pamela J umper Thurman, Ph.D. 
Senior Research Scientist 
Tri Ethnic Center for Prevention Research 
Colorado State University 
Sage Hall 
Ft. Collins, Colorado 80523 
Phone: 800 835-8091 
Email: pjthurman@aol.com 
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Footnote 

^This list is a reiterated and expanded version of the one initially presented 
by Freeman, I ron Cloud - Two Dogs, Novins, and LeMaster (2004), p. 17. 
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